Internal Validity

the degree to which the results of the study are accurate for the participants in the trial

External Validity

the degree to which the results of the study are accurate for patients receiving the treatment in routine care

Good advice for critical appraisal:

Read the study and think about it.

First: Determine the PICO for the study.

Second: Methods

1. Was the assignment of patients to treatments randomized?

2. Were the groups similar at the start of the trial?

3. Aside from allocated treatment, were groups treated equally?

4. Were all patients who entered the trial accounted for?—and were they analysed in the groups to which they were randomized?

5. Were measures objective or were the patients and clinicians kept “blind” to which treatment was being received?

Third: Results

1. How large was the treatment effect? (How was the primary endpoint and important second endpoints measured?)

2. How precise was the estimate of the treatment effect?

Fourth: External validity

1. Will the results help me in caring for my patient?

CURE study (The CURE Investigators 2001)

Was assignment of patients randomized? (1a)

CURE study

After we had obtained written informed consent, patients were randomly assigned to either the clopiogrel group or the placebo group by a central, 24-hour, computerized randomization service. Permuted-block randomization, stratified according to clinical centre, was used.

The CURE Investigators (2001, 495)

Random allocation

• Random allocation requires a random number generator

• Random allocation \(\neq\) haphazard allocation, alternate allocation, etc.

• The unit of random allocation should be the unit of analysis

Block and stratified random allocation

Block random allocation

Allocate participants within a block to ensure similar sizes in each group, e.g. ABBA, BAAB, …

Stratified random allocation

Ensure even numbers of participants with a specific characteristic are randomized to each group. Participants are put in stratified groups and then randomized in blocks.

Were the groups similar at the start of the trial? (1b)

Assessing similarity at baseline

• More important with smaller trials

• The larger the trial, the less likely there will be clinically important differences at baseline

• Crude statistical comparisons at baseline are not reliable

• For the difference to be important, the covariate needs to be a possible confounder (and not adjusted for in the analysis)

Aside from the allocated treatment, were groups treated equally? (2a)

CURE Study

A loading dose of clopidogrel (300mg orally) or matching placebo was administered immediately, followed by clopidogrel (75mg per day) or matching placebo for 3 to 12 months (mean duration of treatment, 9 months). Aspirin (recommended dose, 75mg to 325mg daily) was started or continued simultaneously with the study drug. Follow-up assessments occurred at discharge, at one and three months, and then every three months until the end of the study.

The CURE Investigators (2001, 495)

Equal treatment within the study

• Treatment and follow-up needs to be the same or the differences could bring in biases in some endpoints

• This can be challenging when the intervention is not a medicine (e.g. cognitive behavioural therapy, acupuncture, ‘mindfulness’, physiotherapy, …)

Were all patients who entered the trial accounted for?—and were they analysed in the groups to which they were randomized? (2b)

Information regarding drop-outs is limited

A total of 21.1 percent of patients in the clopidogrel group discontinued the study medication permanently, as compared with 18.8 percent in the placebo group.

The CURE Investigators (2001, 498)

All analyses were based on the intention-to-treat principle…

The CURE Investigators (2001, 495)

Participant flow

• How many participants discontinued their allocated treatment?—why?—what did they receive instead?

• Important to consider flow out of and into the treatment group

• How were patients that discontinued or lost to follow-up analysed?

CONSORT Diagram

Intention-to-treat analysis

Intention-to-treat analysis (ITT)

compares treatment outcomes between participants according to treatment allocation (regardless of whether they received the treatment)

Per protocol analysis

A per protocol analysis compares treatment outcomes among participants who adhered to the treatment protocol

• Random allocation ensures that the treatment groups are comparable at baseline

• What happens during the trial is another matter:

  • Drop outs: participants allocated treatment who discontinue their allocated treatment

  • Drop ins: participants allocated placebo or control who start taking the experimental treatment

• The effects of drop out/drop in/loss to follow up is unlikely to be random

Non-adherence to trial allocation

Starting assumptions

1. The baseline risk of the event in the population is 10%
2. The intervention treatment reduces the relative risk of the event by 20%

Consider the following scenario

1. Non-adherence to treatment allocation occurs equally in both groups
2. High risk drop-in: in the control group, participants with a higher baseline risk “drop in” to the treatment group

Heart Protection Study (Heart Protection Study Collaborative Group 2002)

• The Heart Protection Study randomized \(>\) 20,000 participants with coronary artery disease or diabetes to simvastatin 40mg daily or placebo (starting 1994)

• Participants allocated simvastatin or placebo could start ‘non-study’ statin if GP thought it indicated

• Primary endpoint: all-cause mortality (analysed according to intention-to-treat); 12.9% simvastatin v 14.7% placebo

• Of the participants allocated simvastatin: 82% remained adherent, 3% took a non-study statin, 2% took both

• Of the participants allocated placebo: by year 5, 32% took a non-study statin

WHI calcium study

• Jackson et al. (2006) randomized 36,000 women (50–79 years old) to calcium and vitamin D supplementation for approx. 7 years

• Primary end-point analysed according to intention-to-treat: hip fractures hazard ratio 0.88 (95% CI 0.72–1.08)

• But also provide a pre-planned sensitivity analysis in the adherent population (censored 6 months after identification of nonadherence)

• Approx. 60% of participants were adherent (defined as taking at least 80% of study medications)

• Hip fracture rate among adherers: hazard ratio 0.71 (95% CI 0.52–0.97)

Were measures objective or were the patients and clinicians kept “blind” to which treatment was being received? (3)

CURE Study

We undertook a randomized, double-blind, placebo-controlled trial…

The CURE Investigators (2001, 494)

All primary outcomes and life-threatening and major bleeding complications were adjudicated by persons who were unaware of the patient’s treatment-group assignments.

The CURE Investigators (2001, 495)

Masking

• If participants, clinicians, people involved in collecting data and/or assessing outcomes know what treatment the participant has received it can influence how they volunteer, describe and/or interpret outcomes

• Masking is especially important for subjective outcomes (Hróbjartsson and Gøtzsche 2010)

• Subjective outcomes includes those that rely on participant self-report (e.g. pain, nausea)

Homework!

Reference